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Abstract 


Florida enacted legislation in 2008 for a statewide program known as the Florida 
College and Career Readiness Initiative (FCCRI), which was intended to reduce the need 
for postsecondary remediation. The FCCRI consisted of testing grade 11 students to 
determine their college readiness and offering math and English college readiness and 
success (CRS) courses in grade 12 for students who did not test college-ready the year 
before. FCCRI is based on the theory that providing college readiness testing and CRS 
courses in high school may raise students’ awareness of their academic deficiencies 
and motivate them to further develop college-level skills in their senior year. We found 
considerable variation across districts and schools in the initiative’s implementation 
and level of compliance with state requirements for participation. 


We estimated program impacts using two different methods. First, we used a 
regression discontinuity design to compare outcomes for students scoring just above 
and below test score cutoffs for assignment to the FCCRI. Among the two cohorts of 
students required to participate in the FCCRI, we found little to no effect on short- 
term outcomes including high school graduation, college enrollment, and enrolling in 
or passing nondevelopmental courses. There is little evidence for improved enrollment 
or pass rates in for-credit coursework among the highest- and lowest-performing 
targeted students. However, enrollment and pass rates in transition and degree credit 
courses were similar for students on the margins of assignment to college readiness 
courses in either subject, indicating that students just below college-ready were able 
to “catch up” by the time they enrolled in college. There were also few differences 
between marginal targeted and non-targeted students on longer-term outcomes 
including persistence, transfer, nondevelopmental enrollment and pass, and degree 
completion rates. 


Second, we examined the effect of offering the FCCRI to students from a wider range 
of academic performance levels by using regression analysis to compare student 
outcomes for targeted students in schools before and after the schools implemented 
the FCCRI. As with the regression discontinuity analyses, we found that the FCCRI had 
little to no effect on most short-term student outcomes. We did find that the treatment 
group was more likely to both take and pass nondevelopmental courses in math and 
English. Although the average effects were small, the magnitudes of these effects were 
quite large for some portions of the achievement distribution, with effects of up to 
10.7 percentage points for the treatment group. 
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Finally, we estimated the cost for the FCCRI’s ongoing implementation at $57 per 
targeted student in the 2014/2015 school year, with about 63 percent of per-student 
program costs incurred at the school level ($36), 33 percent at the district level ($19), 
and 3 percent at the state level ($2). We also found that while net costs exceeded 
benefits for the FCCRI as a whole, program effects and costs varied both within and 
between districts. 
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Final Summary report 


There is nationwide concern about the number of students who leave high school 
unprepared for college-level coursework and the lack of student awareness of their 
own levels of preparation. In 2010, approximately 86 percent of community college 
students believed that they were academically prepared for college, yet 67 percent 
tested into developmental coursework (Center for Community College Student 
Engagement, 2016). Results were similar in Florida, where 63.6 percent of first-time 
degree-seeking students at two-year state colleges did not meet the college-ready 
entry-level scores in at least one subject on the placement test and were required to 
enroll in developmental education (Florida College System, 201 2a). 


Florida attempted to address these concerns by implementing the Florida College and 
Career Readiness Initiative (FCCRI), a statewide initiative that provided college 
placement testing to grade 11 students who were mid-performing on the grade 10 
Florida Comprehensive Achievement Test (FCAT), and college readiness and success 
courses (CRS) in grade 12 to students who scored below the threshold for college-ready 
on the placement test in math and reading. Student participation in both the college 
readiness testing and the courses was voluntary when the initiative began in the 
2008/2009 school year. Legislative changes in 2011/12 required college readiness 
testing in grade 11 for mid-performing students on the grade 10 state assessment and 
CRS course participation in grade 12 for students who do not test college-ready. 
Additional legislative changes in 2015 made participation in the FCCRI voluntary once 
again. The FCCRI is based on the theory that providing college readiness testing and 
CRS courses in high school may raise students’ awareness of their own academic 
deficiencies and motivate them to further develop college-level skills in their senior 
year. The CRS courses were designed to develop these skills by improving the 
alignment between the content taught in grade 12 courses and first year college 
courses in math and English. 


Implementation 


When the FCCRI began in 2008/09, student participation in both college readiness 
testing and the CRS courses was voluntary. In 2011/12, participation in both 
components became mandatory for targeted students. The mandatory FCCRI 
continued through 2014/15. This report includes findings for the three cohorts of 


CNA 


Figure 1. Percentage of schools offering students in the voluntary FCCRI 
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schools did not always adhere to 
student eligibility criteria in the state policy. Between 58 to 73 percent of students 
(depending on cohort and subject area) in the mandatory FCCRI were targeted to take 
the Postsecondary Education Readiness Test (PERT) based on their FCAT scores. 
Between 58 and 65 percent of students who scored below college-ready on the PERT 
enrolled in a CRS course in grade 12. Some students took the PERT and enrolled in the 
CRS courses even though they were not required to participate under the mandatory 
FCCRI. Their participation rates ranged from 5 percent to 25 percent, depending on 
FCAT level and cohort. 


so 


0% + 


The content and rigor of the CRS courses varied substantially across schools. In some 
schools, the CRS courses were very similar to the courses taken by students prior to 
the FCCRI. In other schools, the CRS courses placed more emphasis on developing 
academic skills for college and testing college-ready. 


In both math and English, targeted students who scored just above college-ready were 
more likely to enroll in honors and college credit-bearing courses than students who 
scored just below college-ready. However, we also found a small increase (3.5 
percentage points) in the likelihood of taking no math in the senior year for students 
who scored just above college-ready on the PERT in grade 11. 
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Effect of the FCCRI at program cutoffs 


We examined the effect of assignment to take college readiness assessments or 
coursework. Our sample consisted of first-time grade 11 students in 2011/12 (cohort 
M1) and 2012/13 (M2). We used regression discontinuity (RD) analysis to compare 
students just above and below the FCCRI’s cutoffs on the FCAT and the grade 11 PERT. 
If students on either side of a cutoff have similar background characteristics and do 
not bunch on either side, RD will capture the effect of the FCCRI without any outside 
effects. However, it applies only to students very near the cutoff being studied. 


Assignment to participate in the FCCRI had no effect on high school graduation or 
college enrollment. It had no effect on for-credit enrollment or passing in math in 
cohort M1; M2 had too little data for math. In English, students at the low FCAT cutoff 
in M1 were 4.3 percentage points more likely to enroll in for-credit coursework but no 
more likely to pass; there was no effect at the high cutoff (figure 2). In cohort M2, there 
was no effect at the low FCAT cutoff, but students at the high cutoff were 2.8 
percentage points more likely to pass. 


There may be small or null effects at FCAT cutoffs because they capture a wide range 
of abilities. Students at the low cutoff were not yet eligible for high school graduation, 
while almost all of those near the high cutoff were already college-ready. Students 
unprepared to graduate high school likely need more than one course to become 
college-ready, while those who are prepared do not need any. 


The FCCRI may have helped students near the grade 11 PERT’s college readiness cutoff, 
as those targeted for CRS courses performed comparably in college to those who were 
college-ready in grade 11. Since scores below this cutoff corresponded to remediation 
and those above it were exempt, we would expect higher remediation rates just below 
the grade 11 cutoff. Yet in math, students just below and just above this cutoff were 
equally likely to enroll in or pass for-credit courses. In English, students just below the 
cutoff were 2.3 percentage points less likely to enroll in for-credit courses but no less 
likely to pass. In cohort M2, students below this cutoff were less likely to enroll in for- 
credit courses but were no less likely to pass, suggesting that the FCCRI was preventing 
students from failing for-credit courses rather than helping them succeed. For 
comparison, scoring below college-ready after grade 11 had a very large effect for 
cohort M1 on enrolling in or passing a for-credit course within a year of high school 
graduation (remediation was optional for cohort M2). 


Results for students assigned to CRS courses are striking when considering other 
course options. Students who barely scored college-ready on the grade 11 PERT were 
more likely to enroll in honors and other advanced courses during grade 12, while 
those just below the cutoff were much more likely to enroll in standard-level CRS 
courses. Either students assigned to CRS courses fared as well in college as their peers 


in rigorous high school courses or their peers may have enrolled in inappropriately 
challenging courses or ones not aligned with college content. 


We found little long-term effect two or three years after high school graduation for 
students near the FCAT cutoffs. The only effect at any of the FCAT cutoffs for cohort 
M1 was a slight increase in degree receipt within two years for students near the low 
reading cutoff. At the low FCAT reading cutoff, seamless enrollees from cohort M2 
were slightly more likely to persist to a second year of school and students overall 
were more likely to be enrolled in a two-year institution. 


There were long-term effects at the grade 11 PERT cutoffs, though they were small in 
magnitude. In cohort M1, seamless enrollees barely targeted for CRS math courses 
were slightly less likely to transfer from a two-year institution to a four-year one in 
their second year. Seamless enrollees barely assigned to CRS English were slightly more 
likely to pass a for-credit English course within two years and to persist to a third year 
of college. Overall, students at this cutoff were slightly more likely to be enrolled at a 
four-year institution within three years. In cohort M2, students narrowly assigned to 
CRS math were slightly less likely to transfer in year 2 and those assigned to CRS 
English courses were less likely to enroll in a four-year school. 


Figure 2. Effect of the FCCRI on short- and long-term outcomes. 
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Effect of the FCCRI beyond program cutoffs 


We used a different strategy to assess the effect of the FCCRI beyond program cutoffs 
because it is reasonable to believe the effect could be different for students who were 
not near one of the cutoffs. To control for differences in the types of students that 
participate in the FCCRI, we created a sample using a before-and-after design. The 
comparison group included students who attended schools during a low-compliance 
period (less than 5 percent of eligible students participated in college readiness testing 
and CRS courses), while the treatment group included students who attended a school 
during a high-compliance period (at least 50 percent of eligible students participated). 
We limited the sample to schools that switched from low to high compliance to control 
for differences in school-level characteristics that may be related to compliance with 
the FCCRI. We used regression analysis with this sample of comparable students and 
schools and controlled for student background characteristics and pretreatment 
achievement. 


As with the RD analysis, we found the FCCRI had no effect on high school graduation 
or college enrollment. There was, however, an effect on college course outcomes. 
Among targeted students who seamlessly enrolled in college, the treatment group was 
5.7 percentage points more likely to enroll in a nondevelopmental math course, 4.3 
percentage points more likely to enroll in a nondevelopmental English course, and 3.5 
percentage points more likely to pass nondevelopmental courses in both subjects. 


We also broke down the results for the college course outcomes by pretreatment 
achievement, as measured by the continuous variable for grade 10 FCAT score, to 
determine whether there is variation in the effect of the FCCRI. The results are 
presented in Figure 3, which contains predicted probabilities of enrolling in each 
course level by treatment status and baseline achievement. The difference between the 
two lines is the marginal effect of the intent to treat. The FCCRI had a larger effect on 
students in the lower and middle range of baseline achievement. The largest difference 
across treatment status in math was for students near the FCAT level 3 cutoff. In 
English, the largest difference was for students at the bottom of the targeted range. 


As a final line of inquiry, we broke down course enrollment outcomes to include more 
options so we could estimate the impact of the FCCRI on a wider variety of course 
levels (lower level developmental, upper level developmental, transitional (math only), 
and degree credit). The FCCRI reduced developmental enrollment in math. The effect 
was especially large for students at the lower end of the targeted range (FCAT level 2), 
where the treatment group had up to a 12.6 percentage point decrease in the likelihood 
of enrolling in lower-level developmental education courses. Most of this difference 
was due to students being more likely to enroll in transitional math; however, there 
was also an increase in the likelihood of taking no math course in the first year of 
college. The largest overall positive effect was among treated students in the middle 
of the targeted range (FCAT level 3), where there was a 10.7 percentage point increase 
in the likelihood of enrolling in nondevelopmental (transitional or degree credit) 
courses. 


The change to degree credit course enrollment in math was small across all FCAT 
levels. This is not surprising, as students were targeted for CRS courses if their test 
scores would have placed them into a developmental course. Students with placement 
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scores at the transitional level would be exempt from taking a CRS course; thus, the 
FCCRI is trying to move students from developmental courses to the transitional level. 


Developmental education course enrollment was also lower for the treatment group in 
English, though the differences were smaller. The main change was in moving FCAT 
level 2s from upper-level developmental education to degree credit courses, where 
FCAT level 2 treatment students had up to a 5.8 percentage point decrease in the 
likelihood of enrolling in upper-level developmental courses. Over 80 percent of FCAT 
level 3s in both the treatment and comparison groups enrolled in degree credit 
courses, which suggests that many of these students were college-ready by the time 
they enrolled in college, regardless of whether they received any intervention. 


Figure 3. Predicted probability of enrolling in and passing nondevelopmental 
courses by baseline achievement 
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Cost- benefit analysis 


Our analysis aimed to determine whether the FCCRI’s benefits outweighed its costs. 
We surveyed state, district, and school personnel to estimate the cost of the FCCRI 
based on the ingredients used in its implementation. We estimated benefits based on 
RD analysis (see “Effect of the FCCRI at program cutoffs”). Due to data demands, we 
aggregated benefit estimates at the district level, not at individual schools. 


The annual cost of the FCCRI, not including startup costs, was approximately $57 per 
student (Table 1). With an estimated 128,988 students from cohort M1 participating, 
the total cost of all resources devoted to the FCCRI for its students was approximately 
$7 million. However, since most of these resources were repurposed from other uses, 
the actual on-budget cost of the FCCRI consisted only of state reimbursements to 
schools and districts for administering the PERT and totaled only about $150,000 per 
year, or just over $1 per student. 


About 63 percent was incurred at the school level, with district-level costs constituting 
about 33 percent and state-level costs about 3 percent. Personnel costs, incurred 
mainly at the school level, were the most expensive component of the program, at 
about 69 percent of the total. Materials and equipment made up about 30 percent of 
program costs, and were incurred mainly at the district level. Facilities made up only 
a very small proportion of FCCRI implementation costs. CRS costs represented about 
26 percent of total program costs, while PERT testing costs were about 74 percent. 


Per-student costs varied across schools and districts. Across schools, estimated total 
costs (including state- and district-level costs) ranged from $23 to $312 per student. 
Schools and districts that allocated more resources to the FCCRI tended to dedicate 
them to PERT administration; providing teachers with professional development; and 
CRS course materials, such as planning guides, textbooks, and computer software. 
Schools that tested relatively few students tended to have higher per-student costs 
and may have had difficulty achieving economies of scale. 


The FCCRI produced a range of different benefits, which we define as the reduction in 
required costs prior to passing a for-credit course, based on students’ preparation 
levels and the districts in which they were enrolled. Of the eight districts that met our 
sample requirements, three did not produce any discernible benefits and two others 
produced negative benefits (Figure 4). Of the three with any positive benefits at any 
cutoffs, only one had positive benefits at more than one cutoff. 


When comparing expected benefits against average costs, six of the eight districts had 
negative net benefits at every cutoff, and the other two had negative net benefits at 
most cutoffs. This would not change if we instead used the lowest per-student costs 
from our cost sample; three sets of school-level costs would yield negative net benefits 
at all cutoffs. Thus, while a program such as the FCCRI can be implemented extremely 
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cheaply if substantial resources are repurposed from other uses, additional on-budget 
costs could be necessary to see widespread net benefits. 


Table 1. Perstudent annual cost of the FCCRI, by cost ingredient 


PERT testing CRS courses 
Cost Total 
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* Estimated cost, rounded to the nearest dollar, is less than $1 per student. 


Figure 4. Net benefit of the FCCRI, using average perstudent cost of $57 
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Note. Values are in 2014 dollars and rounded to the nearest dollar. Values in bold apply to 
all cutoffs for a district. Districts and cutoffs with too few students (including the low FCAT 
cutoff in math) are omitted. 
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Conclusions 


Our findings and the lessons learned about the initiative’s design and implementation 
may have implications for researchers and policymakers considering similar college 
readiness programs. These findings may also help to explain why the FCCRI had few 
discernable effects on student outcomes. 


Lessons learned about the design of the FCCRI: 


e The FCCRI did not seem to adequately consider students’ motivations to attend 
college. The initiative was focused on helping students become college-ready 
regardless of whether they intended to enroll in college. No additional advising 
or support was provided to non-college-bound students to improve their 
career options or to encourage them to consider other postsecondary 
programs. 


e The FCCRI likely targeted students from too wide a range of achievement 
levels. Some higher-performing students took CRS courses at the expense of 
more rigorous grade 12 courses, while lower-performing students may have 
needed more than a single course to become college-ready. A single course 
cannot meet the needs of every student from such a wide range of achievement 
levels. 


e The FCCRI was largely an unfunded mandate, resulting in varying reallocations 
of existing resources among schools and districts. This disparity led to uneven 
implementation across districts and schools, some of which may not have had 
the resources to serve all targeted students adequately. 


e While the Florida Department of Education provided funding for college 
readiness testing and approved CRS courses, this may not have been enough 
to successfully implement the initiative. Teachers and administrators lacked 
guidance beyond broad standards on what CRS courses should over or how 
they would do so. This confusion likely contributed to lower and less-uniform 
instructional quality that lacked alignment with the original intent of the 
initiative. 


e It may have been more effective to begin participation in the FCCRI in earlier 
grade levels. The Institute of Education Sciences (IES) Practice Guide on 
pathways to college suggests offering college readiness assessments 
throughout high school and recommends that students begin preparing for 
college-level work by ninth grade (Tierney, Bailey, Constantine, Finkelstein, and 
Hurd, 2009). 
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Lessons learned about the implementation of the FCCRI: 


e Schools and districts often did not follow state requirements for participation 
in the FCCRI; this situation may have been exacerbated by the lack of 
enforcement or sanctions for non-compliance. Under the voluntary FCCRI, 
most schools did not offer college readiness testing, CRS courses, or both. 
Although school-level compliance increased under the mandatory FCCRI, many 
targeted students did not participate. 


e Schools required more guidance and better communication about how to 
implement the FCCRI, as teachers were often unaware of the content of the 
PERT or of how CRS courses were supposed to prepare students for college- 
level work. While FLDOE recommended that high schools collaborate with local 
colleges, we found very little evidence of collaboration around CRS courses or 
college readiness more broadly. The few educators who did participate in 
cross-sector collaboration indicated that they found it very helpful, and those 
who did not participate expressed a desire for more opportunities to do so. 


e Though CRS courses shared the same names and course numbers statewide, 
they were implemented differently across and sometimes even within schools. 
This made it difficult to evaluate the program as a whole. CRS courses will have 
little effect if they do not contrast with existing courses (e.g., some English CRS 
courses covered the same material in the same way as the regular English 4 
courses). 


